System and method for generating questions and multiple choice answers to adaptively aid in word comprehension

ABSTRACT

An adaptive learning system and method provides for automatically generating question types to a user for word comprehension and selecting multiple choice answers for display. Questions are developed for the user by obtaining online content and indexing the content into individual sentences and questions. The system provides questions in a series of rounds to the user and then adaptively tracks the progress of the user based on the categorization of each question.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit, under 35 U.S.C. §119(e), of U.S.Provisional Patent Application No. 61/324,136, filed on Apr. 14, 2010,which is herein incorporated by reference in its entirety.

FIELD OF THE INVENTION

The present invention relates to a system and method for automaticallygenerating various question types, including automatic selection ofmultiple choice answers for display. The present invention furtherrelates to a system and method for selecting presentable multiple choiceanswers based on use of a word in a sentence, quality of a sentence, andfrequency of use of the word in other sentences. The present inventionfurther relates to an adaptive learning system which aids a user in wordcomprehension by asking questions in a series of rounds and thentracking the progress of the user based on the categorization of eachquestion.

BACKGROUND INFORMATION

Particularly effective methods for improving grammatical skill includehaving an individual actively complete sentences by filling in blankportions of the sentences or be tested on the definition, synonym,and/or antonym of a given word. Such activities are also often used totest grammatical skill. For example, a user, such as a test taker, maybe provided with a set of possible choices of words for selection tofill in the blank portion of, and thereby complete, the sentence. Suchfill-in sentences are currently manually compiled, which entails atedious process.

Additionally, a test taker can be asked a number of questions aboutgiven words, including, to provide a definition for a given word basedon a list of choices for definitions, or providing a synonym or antonymfor that word. After a test taker answers a question, the test takermoves on to answer a new question about another word. Although a testtaker may answer a question correctly, the test taker might not fullyunderstand the definition or etymology of a word, or might have guessedto arrive at a given answer. Thus, in these antiquated tests orprograms, a test taker may be given a false sense that the test takerfully understands a word, when in fact the test taker does not. A testtaker is not provided the opportunity to be subsequently tested on agiven word after test completion, to ensure that the test takerunderstands all definitions and uses of the given word, and has masteredknowledge of a word. Further, a test taker does not have an opportunityto adapt questions asked based on the test taker's level of vocabularycomprehension and ability.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an adaptive learning system, according toan example embodiment of the present invention.

FIG. 2 is a flow diagram of a process of generating questions andmultiple choice answers from retrieved text, according to an exampleembodiment of the present invention.

FIG. 3 is a flow diagram of a process of determining questions andmultiple choice answers in accordance with a determined questioncategory, according to an example embodiment of the present invention.

FIG. 4 is a graph plotting ability categories against a percentage ofrespondents of groups who have correctly responded to a fill-in sentencetype question, according to an example embodiment of the presentinvention.

FIG. 5 is a screen shot of an interface of an adaptive learning system,according to an example embodiment of the present invention.

FIG. 6 is a flow diagram of a process of determining an allocation ofquestions by question category in each of a plurality of rounds,according to an example embodiment of the present invention.

DETAILED DESCRIPTION

Example embodiments of the present invention provide a vocabularylearning and testing environment to facilitate word and vocabularycomprehension, in which a system and/or method automatically generatesquestions and answer choices from designated sentences or words, andadapts future outputted questions based on selections of answer choicesby the user/test-taker.

In example embodiments of the present invention, a system and methodprovides for an adaptive learning system where questions to a user maybe adapted to an individual user, by asking a user questions in a seriesof rounds and then tracking the progress of the user based on thecategorization of each question.

According to an example embodiment of the present invention, a systemand method automatically compiles partially blank fill-in sentenceswhich may be used, for example, to hone and/or test grammatical skill.

According to an example embodiment of the present invention, a systemand method automatically selects for output a set of possible textstrings, each including one or more words, which may be selected by auser for completing partially blank fill-in sentences.

According to an example embodiment of the present invention, a systemand method automatically selects for output a set of possible textstrings, each including one or more words, which may be selected by auser to indicate that it is a synonym of a designated word displayed tothe user.

According to an example embodiment of the present invention, a systemand method automatically selects for output a set of possible textstrings, each including one or more words, which may be selected by auser to indicate that it is an antonym of a designated word displayed tothe user.

According to an example embodiment of the present invention, a systemand method automatically selects for output a set of possible textstrings, each including one or more words, which may be selected by auser to indicate that it is a definition of a designated word displayedto the user.

According to an example embodiment of the present invention, a systemand method provides a sentence including a designated word displayed tothe user and automatically selects for output a set of possible textstrings, each including one or more words, which may be selected by auser to indicate that it is a synonym of the designated word in thecontext of the sentence.

According to an example embodiment of the present invention, a systemand method provides a sentence including a designated word displayed tothe user and automatically selects for output a set of possible textstrings, each including one or more words, which may be selected by auser to indicate that it is a definition of the designated word in thecontext of the sentence.

FIG. 1 illustrates a diagram of a terminal 10 displaying a userinterface of adaptive computer learning program 20 stored in a memory15, accessible by a processor 30, according to an example embodiment ofthe present of the present invention. Adaptive learning program 20 maybe executed by processor 30 and may result in an output to be displayedon terminal 10 to a user. Terminal 10 may be a computer monitor, or anyother display device which may depict adaptive learning program 20during execution.

Processor 30 may be implemented using any conventional processingcircuit and device or combination thereof, e.g., a Central ProcessingUnit (CPU) of a Personal Computer (PC) or other workstation processor,to execute code provided, e.g., on a hardware computer-readable mediumincluding any conventional memory device, to perform any of the methodsdescribed herein, alone or in combination. Processor 30 may also beembodied in a server or user terminal or combination thereof.

The components of FIG. 1 may be embodied in, for example, a desktop,laptop, hand-held device, Personal Digital Assistant (PDA), televisionset-top Internet appliance, mobile telephone, smart phone, etc., or as acombination of one or more thereof. The memory 15 may include anyconventional permanent and/or temporary memory circuits or combinationthereof, a non-exhaustive list of which includes Random Access Memory(RAM), Read Only Memory (ROM), Compact Disks (CD), Digital VersatileDisk (DVD), and magnetic tape.

An example embodiment of the present invention is directed to one ormore hardware computer-readable media, e.g., as described above, havingstored thereon instructions executable by processor 30 to perform themethods described herein.

An example embodiment of the present invention is directed to a method,e.g., of a hardware component or machine, of transmitting instructionsexecutable by processor 30 to perform the methods described herein.

Various methods and embodiments described herein may be practicedseparately or in combination.

FIG. 2 is a flowchart that illustrates an example process of generatingquestions and multiple choice answers from retrieved text. In an exampleembodiment, in step 100, one or more servers may obtain text fromInternet content and generate questions based on sentences from thetext. For example, for fill-in sentence question types, these sentencesmay be made by removing, for completion by a user, a portion of each ofone or more of sentences of the obtained text. For synonym, antonym, anddefinition type questions, individual words may be extracted from thetext. Synonym hint and definition hint questions may provide anunobstructed sentence which may provide an unclear meaning of ahighlighted word in a provided sentence.

For example, the server(s) may be subscribed to web content syndication(RSS) feeds, such as RDF Site Summary, Rich Site Summary, or ReallySimple Syndication feeds, of major newspapers and periodicals, and mayuse text from such feeds for generating the fill-in sentences and othersentence questions.

Internet articles obtained, for example from RSS feeds, typically takethe form of a Hyper Text Markup Language (HTML) document. In an exampleembodiment of the present invention, the system and method parses theHTML document into an eXtensible Markup Language (XML) Document ObjectModel (DOM), including a hierarchy of nodes and attributes which may beprogrammatically examined for analyzing the text for boilerplatelanguage. For example, the system and method may compute a respectivehash code for each node of the XML DOM based on the text contained inthe respective node (including the text of all of the node's childnodes), which hash codes may be used for the analysis as described infurther detail below.

In step 110, the system and method may analyze punctuation marks of theobtained text to determine the boundaries of sentences within the text.For example, at least initially, each punctuation mark (even those thatare not usually used to end a sentence, e.g., a comma) may be considereda sentence boundary. In step 120, the system and method may then furtheranalyze words surrounding the punctuation marks and discard thepunctuation mark as a sentence boundary where the surrounding wordssatisfy certain predetermined conditions. For example, where “Mr”precedes a period, the system and method remove that period as asentence boundary. In an example, a further condition may be requiredwith respect to the preceding example that the period also be followedby a proper name for its removal from consideration as a sentenceboundary.

In step 130, the system and method may apply a part-of-speech (POS)tagger to identify the POS (e.g., noun, verb, adjective, etc.) for eachword in each of the obtained sentences, and may store POS tags for eachof the words identifying the respective POS of the word.

The system and method may store the POS parsed sentences in a databaseof one or more indices in step 140. For example, the system and methodmay index the sentences by included words and POS of those words. In anexample embodiment, certain words, such as “a,” “the,” “it,” etc., maybe discarded from use for indexing the sentences. In an exampleembodiment, the indexing of sentences may be by only those words thatare found in, or are associated with, those words found in (as explainedbelow), a designated electronic dictionary.

In an example embodiment, the system and method may look up the indicesfor those sentences which include those certain words, for example,having the POS of those certain words. For example, where the word“store” as a noun (a mercantile establishment for the retail sale ofgoods or services) is selected, the system and method may obtain anindexed sentence including the word store used as a noun, and where theword “store” as a verb (keep or lay aside for future use) is selected,the system and method may obtain an indexed sentence including the wordstore used as a verb.

The system and method may include or provide a user interface, e.g., theuser interface of adaptive learning program 20, by which the system andmethod may receive input of selected words to be tested. Alternatively,the words may be ranked by difficulty, and, for a selected difficultylevel, the system may automatically select, e.g., randomly, words from acorpus assigned the selected difficulty, and then select sentences forthose words, e.g., randomly from a set of highly ranked sentences, whoseranking may be as described in detail below with respect to an exampleembodiment. Alternatively, the questions as a whole may be ranked bydifficulty and/or by ability to discriminate between different skilllevels of test takers, and may be selected based on the actual orexpected skill level of the test taker and/or the ability todiscriminate between skill level on the basis of the sentence.

In an example embodiment of the present invention, the system and methodmay automatically identify portions of the obtained text which isboilerplate language, and may weed those textual portions out, so thatthey are not indexed. For example, the system and method may storeobtained text in a boilerplate database. As new text is obtained, thenew text may be compared to the text in the boilerplate database. Ifthere is a match, the text may be discarded and not indexed. (The textmay remain in the boilerplate database for comparison to later obtainedtext.) In an example embodiment, the system and method may maintain acounter for each textual component of the boilerplate database, andincrement the counter each time a match is found. According to thisexample, the text may be discarded conditional upon that the countervalue is at least a predetermined threshold value.

In an example embodiment, the boilerplate removal may be on a sentenceby sentence basis, so that text is discarded as boilerplate only if theentire sentence meets the conditions for discarding.

In an alternative example embodiment of the present invention, theboilerplate removal may be on a node by node basis (described in detailbelow). For example, a block of text may correspond in its entirety to afirst node, and subsets of the text block may correspond to respectivenodes of lower hierarchical level than the first node, which lower-levelnode may include even lower level nodes, etc. Accordingly, even wherethe boilerplate analysis provides for discarding text corresponding to aparticular node determined to be boilerplate, the text may neverthelessremain in the database of indices as part of a larger portion of text.

In an example embodiment of the present invention, the system and methodmay timestamp each received text. In an embodiment, the system andmethod may condition the discarding of the text on repeated occurrenceof the text within a predetermined period of time. For example, if thesecond occurrence occurred more than a predetermined amount of timeafter the prior occurrence of the text, then the system and method wouldnot discard the text on the basis of that repeated occurrence of thetext. Different time periods may be used for different sources, forexample, depending on the respective frequencies at which text isobtained from the respective sources.

The timestamp may instead or additionally be used as a basis forclearing out stale data from the boilerplate database. For example, iftext initially is obtained once every three days, the text would bestored in the boilerplate database, and whenever new text matching theboilerplate text is obtained, the system and method would refrain fromincluding the text in the indices database, based on its match to thetext of the boilerplate database. If, subsequently, the system andmethod ceases to receive the text, e.g., from the source with which theboilerplate text is associated, for an extended period of time, e.g.,two weeks, measured based on the timestamp of the last receipt of thetext, it may be assumed that the boilerplate text is no longer beingused, e.g., by the source, and the system and method may therefore beconfigured to remove the text from the boilerplate database.

In an example embodiment of the present invention, if the identifiedboilerplate text has already been stored in the indices database, thesystem and method may remove the text from the indices database.

In an example embodiment, for each source, the system and method mayrefrain from storing any of the obtained text in the indices databaseuntil a minimum amount of text or number of articles have been obtainedfrom the source and analyzed for boilerplate. Once a threshold of textor articles have been analyzed for boilerplate, newly obtained textand/or the previously obtained text not identified as boilerplate may beindexed.

In an example embodiment of the present invention, the system and methodmay maintain a separate collection of text in the boilerplate databasefor each different source from which the system and method obtains text.For example, each obtained text may be tagged with a source identifier,e.g., NYT for New York Times, or a separate database may be used foreach different source.

In an example embodiment of the present invention, the system and methodmay perform the boilerplate analysis based on a hash code, ahierarchical level of the text, the identified content source, and/or atimestamp that indicates the previous occurrence of the text. Use of thesource identification and timestamp are described above. The hash codemay be generated for each obtained text block based on the content ofthe text block. For each newly obtained text, the system and method maycalculate a respective hash code and compare the hash code to a set ofhash codes stored in the boilerplate database. The text may bedetermined to be identical to previously obtained text where the hashcodes match. As explained above, this may be done on a source-by-sourcebasis. Accordingly, it may be unnecessary to store the complete text inthe boilerplate database, the hash codes being stored in the boilerplatedatabase instead. Alternatively, the text may be maintained in theboilerplate database at least until it is determined that the text is infact boilerplate text or until enough text has been analyzed to storethe text in the indices database, where the text has not been identifiedas boilerplate text.

In an example embodiment of the present invention, the hash code may begenerated by input of the text into a hashing algorithm. The system andmethod may use any suitably appropriate hashing algorithm, e.g., MD5 orCRC32.

In an example embodiment of the present invention, the matching ofhashing codes may be one of a plurality of factors used for determiningwhether text is boilerplate. Different factors may be given differentweights. For example, an additional factor the system and method mayconsider is the hierarchical position of the node and whether thehierarchical level of the node of the newly obtained text matches or isclose to (determined using a suitably appropriate near-duplicatedetermination method) that of the text of the boilerplate database. Inan example embodiment, the system and method may further generate astring that represents the respective node's unique place in the DOMhierarchy. An example string may be “HTML/Body/P5,” which indicates thatthe text was found in the fifth paragraph of the body portion of an HTMLdocument. The boilerplate text may have occurred, for example at“HTML/Body/P3,” in which case the system and method may determinewhether the new text is boilerplate based on its positional removal fromthe boilerplate text by only two paragraphs.

As noted above, the indexing of sentences in step 140 may be by onlythose words that are found in, or are associated with those words foundin, a designated electronic dictionary. The sentences may includevariations of the words of the dictionary, whose precise form is notincluded in the dictionary. Accordingly, in an example embodiment of thepresent invention, for those sentences which have not been determined tobe boilerplate, the system and method may index the sentence by thosewords of the sentence which are in the electronic dictionary. For thosewords not in the electronic dictionary, the system and method maydetermine whether the words include any of a predetermined set of commonsuffixes. Where a word includes such a suffix, the system and method maystem the word using a stemming algorithm, that may be structured inaccordance with grammatical rules and that may vary by POS, to obtain abase word, which base word the system and method may compare to thewords of the dictionary. For example, the base word may be obtainedmerely by removing the suffix or by removing the suffix and adding aletter. Where the base word matches a word of the dictionary, the systemand method may index the sentence by the base word. In an exampleembodiment, the system and method may do the same for prefixes. In analternative example embodiment, words modified by a prefix may be storedin the electronic dictionary as a separate word independent of the wordwithout the prefix, and the stemming algorithm may accordingly not beapplied to stem prefixes.

In another alternative example embodiment, a combination of automaticquality scoring and manual sorting may be used. For example, the systemand method may automatically assign a quality score. Those sentencesassigned a quality score that does not satisfy a predetermined thresholdquality score are not provided for the manual sort and are therefore notoutput to a user for sentence completion. The automatic scoring may beperformed prior to indexing, and the system and method may refrain fromindexing those sentences assigned a quality score that does not satisfythe predetermined threshold quality score. Those sentences assigned aquality score that does satisfy the predetermined threshold qualityscore may then be output to a reviewer for manual review and assignmentto one of the quality categories.

In an example embodiment of the present invention, the system and methodmay automatically assign a quality score to each of the sentences. Thesentences may be automatically grouped into sentence quality categoriesbased on the assigned quality scores in step 170. For example, eachquality category may correspond to a respective interval of qualityscores.

For example, the system and method may analyze each sentence withrespect to various parameters, which parameters may be assigneddifferent weights in an equation that produces the quality score. Anon-exhaustive list of parameters which may be considered includes thenumber of proper nouns the sentence includes and/or the ratio of propernouns to other nouns or words of the sentence, whether the sentencecontains unbalanced quotes (e.g., an open quotation mark without a closequotation mark), the number of non-alphanumeric characters (e.g.,parenthesis, punctuation, etc.) and/or ratio of such characters toalphanumeric characters, the length of the sentence, whether thesentence ends without a standard ending punctuation mark, whether thesentence begins with character other than a letter or quotation mark,the number of acronyms in the sentence and/or the ratio of acronyms toother words of the sentence, the number of capitalized words and/or theratio of capitalized words to other words of the sentence, and whetherthe sentence begins with a preposition.

For example, the larger the number of proper nouns, the larger thenumber of non-alphanumeric characters, the larger the number ofacronyms, the larger the ratio of proper nouns to other nouns and/orwords of the sentence, the larger the ratio of non-alphanumericcharacters to alphanumeric characters, the larger the ratio of acronymsto other words of the sentence, the larger the number of capitalizedwords, and/or the larger the ratio of capitalized to other words of thesentence, the worse the score may be. Inclusion of unbalanced quotationmarks, a non-standard ending punctuation mark, a beginning characterother than a letter or quotation mark, and/or a preposition as the firstword of the sentence may also reduce the score. The score may also bereduced proportionate to a length by which the sentence exceeds and/orfalls short of a predetermined ideal sentence length.

According to the example embodiment in which sentences are manuallyreviewed, the system and method may produce a large corpus of sentencesto be manually reviewed for quality by a reviewer. In an exampleembodiment, the system and method may therefore prioritize the sentencesin step 150 to be manually reviewed and output the sentences to thereviewer in order of the priorities, so that the most highly prioritizedsentences are reviewed in step 160 and made available for output to auser before sentences of lower priority.

The sentence priorities assigned may be based on priorities of the wordsof the dictionary included in the sentences, such that the higher thepriority of words which a sentence includes, the higher the priority ofthe sentence. Where the highest priority words of two sentences are ofthe same priority, the sentence including the larger number of words ofsuch priority may be ranked higher. Where the number of such words isequal, the next highest priority words of the sentence may be consideredfor prioritizing one of the two sentences ahead of the other. Where allword priorities of two sentences are equal, the sentences may beassigned the same priority values. In alternative example embodiments,other ranking equations may be used for ranking sentences based onpriorities of the words of the sentences. For example, the system andmethod may add the priorities of each sentence and divide the totalpriority value by the number of words or prioritized words of thesentence to obtain an average that the system and method may use.

In an example embodiment of the present invention, the system and methodmay use one or more of the following factors for prioritizing the words,on whose basis the sentences may, in turn, be prioritized: a likelihoodof a word to appear in a standardized test, for example, determinedbased on analysis of a corpus of standardized tests, such as the SAT orGRE, where the higher the likelihood, the higher the priority; how oftena word is looked up on dictionary web sites, where the more the word islooked up, the higher the priority; whether a sentence has already beenmade available for output for a word, where, if a sentence has not yetbeen made available for the word, the word is ranked higher; and whethera sentence has already been made available for a particular sense of theword, where, if a sentence has not yet been made available for theparticular sense of the word, the word, e.g., with respect to theparticular sense, is ranked higher. The likelihood of the appearance ofa word in a standardized test may be manually input into the system.Alternatively, whether a sentence has already been output for review fora particular word or particular sense of the word may be considered.With respect to how often a word is looked up, the system and method maymaintain a dictionary website which may be accessed for looking up themeaning of a word, and may maintain a record of the number of times eachof the words is looked up. Alternatively or additionally, the system andmethod may obtain such records from external dictionary websites.

Based on the priorities of the words, the words may be placed into aqueue. The system and method may sequentially traverse the queue ofwords, and, for each traversed word, search for a sentence associatedwith the word, and, if such a sentence is found, output the sentence forreview. After output of the sentence for review or after review of thesentence, the word may be placed at the back of the queue.

In an alternative example embodiment, the system and method does notsequentially traverse the queue. Instead, position in the queue may beused as a priority factor to be considered along with all other priorityfactors, where the highest priority words are selected.

In an alternative example embodiment, for a particular word for which asentence has been reviewed, the number of other words that have beenreviewed since the review of the sentence for the particular word may beconsidered as a factor for determining the word's priority, and theoverall priority may decide the word's position in a queue.

In an example embodiment, where a word has a number of senses (differentmeanings), and a sentence has been reviewed for only one of theplurality of word/sense pairs of the word, the system and method mayconsider the word as not having been reviewed. Alternatively, that aword has a plurality of word/sense pairs may reduce the impact of areview of a sentence for a single one of the word/sense pairs on thepriority of the word in the queue.

In an example embodiment of the present invention, after all sentencesavailable for a given word are reviewed, the word may be removed fromthe queue, and those words not in the queue may be assigned NULL or itsequivalent for its priority. When new text is obtained that includes theword, the word may then be re-inserted into the queue.

More than one reviewer may review the sentences. The reviewers may usedifferent workstations at which the sentences are output. The system andmethod may divide the sentences to be reviewed between the variousreviewers, e.g., which may be signed into the system, and outputdifferent ones of the sentences to the different workstations at whichthe different reviewers are signed in.

A reviewer may assign a sentence to a quality category, in step 170,such as “excellent,” “good,” or “bad.” For example, a reviewer maydesignate a sentence as being of “excellent” quality if the word appearsin a manner consistent with the word/sense pair, where the word may bedetermined from the context of the sentence. In an example embodiment, areviewer may designate a sentence as excellent if it is used in asentence that provides a context that may clue the reader in on thedefinition of the word. For example, the sentence “Albert applied aliberal amount of suntan-lotion so that it was ensured that every inchof his torso was covered by multiple layers of suntan-lotion” may bedesignated as excellent because it suggests the definition for the word“liberal,” in contrast to, for example, the sentence, “Albert applied aliberal amount of suntan lotion,” which provides less contextualinformation usable as a suggestion of the definition.

FIG. 3 illustrates a process of determining questions and multiplechoice answers to output in accordance with a determined questioncategory. After sentence quality is determined in step 170, questionsmay be provided to the user. For example, a sentenced designated asexcellent may subsequently be designated for use as a fill-in the blankquestion in step 220. Conversely, a sentence designated as “good” instep 170 may not be used as a fill-in the blank question because it doesnot provide sufficient contextual information to suggest the definitionwith the removal of a word. A sentence designated as “good” may be usedto help generate other question types, such as synonym, antonym, anddefinition questions.

A designation of a good classification may be used in instances where aword appears in a way consistent with the target word/sense pair, butthe context of the entire sentence is insufficient to allow for adetermination of the definition of the word. In an example embodiment,the system and method may be configured such that sentences designatedas good are not used as fill-in the blank questions. Such sentences maybe used for other question types where the word is included in theoutput sentence provided for the question, e.g., synonym, antonym, ordefinition questions. The system and method may, in step 230, indicate,e.g., by highlighting, which of the included words is the subject of thequestion.

A reviewer may also designate a question as “bad.” A sentence classifiedas bad may, for example, contain an error or a typo, or may use jargonin the context of the sentence. A sentence classified as bad may alsouse the word in a manner that is inconsistent with the word/sense pairor may use the word according to an incorrect definition. In an exampleembodiment, the system and method may be configured not to use anysentences classified as bad for any of the question types and to discardthe sentence in step 210.

The reviewer may also tag the correct sense, i.e., meaning, of therelevant word in the sentence. For example, for the noun “store” in aparticular sentence, the reviewer may input whether the word, forexample, is intended to mean “a mercantile establishment for the retailsale of goods or services” or “a stock of something,” which tagged sensemay be used for the indexing of the sentence.

Referring again to FIG. 2, and as noted above, the system and method mayprovide for a number of question types in step 180 including fill-insentence questions, questions asking about the synonym of a designatedword, questions asking about the antonym of a designated word, andquestions asking about the definition of a designated word. The systemand method may output a set of multiple choice answers in step 190 fromwhich the user may choose.

For example, the system and method may output a set of multiple choiceanswers in step 190 from which the user may select one for completing afill-in sentence that has been output in step 180.

Referring again to FIG. 3, in an example embodiment, where a fill-insentence question is provided to the user, the system and method mayremove the selected word from a designated sentence in step 221, inserta, e.g., underlined, blank space, and output the modified sentence. Thesystem and method may include the word that had been removed from thesentence, i.e., the correct answer, as one of the answer choices, andmay, in step 222, automatically select a predetermined number of wronganswers for inclusion as the other choices for the fill-in sentence. Ifa user does not answer the question correctly in step 223, they mayencounter the question again in step 224.

In an example embodiment of the present invention, the system and methodmay analyze a large corpus of words, on the basis of which analysis thesystem may select the wrong words for the fill-in the blank question.The analysis may be of factors including, the POS of the word,similarity of meaning to the correct word, similarity of use to thecorrect word, similarity of frequency of use to the correct word, and/orskill level.

For example, the system and method may analyze the corpus of words withrespect to a combination of the above-enumerated factors. For example,in an instance where three incorrect words are presented as choicestogether with the correct word for a fill-in the blank question, thesystem and method may select three words having the same POS as that ofthe correct word, having a meaning dissimilar by a predeterminedquantification to the correct word, a use with similar characteristicsas those of the correct word, a closest frequency of use to the correctword, and a predetermined threshold skill level.

In an example embodiment, with respect to similarity of meanings, thesystem and method may query a corpus of words, e.g., one or more lexicaldatabases, such as WORDNET®, for word/sense pairs that are synonyms orclose relations of the correct word/sense pair, and may exclude resultsof the query indicated to have a very close meaning to that of thecorrect word/sense pair from the set of wrong answer choices for thefill-in the blank questions.

For example, the lexical database(s) may indicate the degree by whichword/sense pairs are related. For example, the lexical database(s) mayindicate whether a word/sense pair is a synonym of, similar to, relatedenough as a “see also” reference of, or belongs to a domain of anotherword/sense pair or is the domain to which the other word/sense pairbelongs. Moreover, the lexical database(s) may use a series of pointersfrom word/sense pair branching out a number of levels from word/sensepair to word/sense pair. In an example embodiment for a fill-in theblank question, for each level traversed from the correct word/sensepair to other word/sense pairs via the pointers of the lexicaldatabase(s), the system and method may assign a cost to the move, wherethe cost depends on the defined relationship of the pointer. Forexample, a synonym may have a cost of zero or nearly zero, while a “seealso” relationship may have a cost of 3. Beginning at the correctword/sense pair, the system and method may traverse the pointers alongvarious branches from level to level until, for a respective branch, athreshold cost is reached, at which point the system and method maycease further traversal along that branch. All traversed word/sensepairs may be eliminated from being a possible wrong answer choice.

In an example embodiment, with respect to similarity of use, the systemand method may apply different conditions for inclusion of a word/sensepair as one of the wrong answer choices, depending on the POS of theword/sense pair.

For example, if the correct word is tagged as a noun, the system andmethod may analyze metadata associated with the correct word todetermine whether it is countable, such as “people,” “chairs,” and“files,” or uncountable, such as “esteem” and “water,” and may requireall wrong answer choices to have the same countability characteristic asthat of the correct word/sense pair. The metadata indicating thecountability of the word may be manually entered into the system (orobtained from an external database). The system and method may furtherrequire that all wrong answer choices share the same set of uniquebeginners as that of the correct word, which is a noun. For example,WORDNET® classifies every noun as having a set of one or more uniquebeginners, i.e., as belonging to a set of one or more ontologicalcategories, a non-exhaustive list of which includes {act, activity},{artifact}, {attribute}, {cognition, knowledge}, {communication},{event, happening}, {feeling, emotion}, {group, grouping}, {location},{natural object}, {person, human being}, {process}, {relation}, {state},and {substance}, where each of the sets of the listed sets of bracketsis a different ontological category. The system and method may query thelexical database(s) for, and compare, the set of unique beginners forthe correct word and the sets of unique beginners for other nouns of thedatabase(s).

For example, if the correct word is tagged as a verb, the system andmethod may require that all wrong answer choices share the same set ofverb frames as that of the correct word. For example, WORDNET®associates each word with a set of one or more verb frames. A verb frameis a phrase structure to which a verb can be applied. For example, averb frame may be “an object does something to a person.” The word“kill” is one of a plurality of verbs that can be applied to such astructure because, for example, a bullet kills a person. Thus, forexample, a verb frame may represent whether a verb is transitive orintransitive or whether the verb applies to people and/or things. Thesystem and method may query the lexical database(s) for, and compare,the set of verb frames for the correct word/sense pair and the sets ofverb frames for other verbs of the database(s).

For example, if the correct word is tagged as an adjective, the systemand method may require that wrong answer choices share the attributionalproperty as that of the correct word. For example, if the correct wordis a predicative adjective, the system and method may require that thewrong answer choices be predicative adjectives as well, and, if thecorrect word is an attributive adjective, the system and method mayrequire that the wrong answer be an attributive adjective as well. Thesystem and method may query the lexical database(s) for, and compare,the attributional properties of the correct word and of other adjectivesof the database(s).

Other rules may be applied for other parts of speech. Alternatively, thesystem and method may output sentences for only word/sense pairs thesystem may group as belonging to or being similar to one of the aboveenumerated parts of speech. For example, some adverbs may be treated asadjectives.

In an example embodiment, with respect to similarity of use, for acorrect word that has been modified from a version in the electronicdictionary by inclusion of a suffix (and/or prefix), the system andmethod may additionally require all wrong answer choices to be able toaccept a similar suffix (and/or prefix). For example, if the correctword has been pluralized, the system and method requires all wronganswer choices to be capable of being pluralized, but not necessarily inthe same way. For example, “wolf” is pluralized by substituting “yes”for “f,” while desk is pluralized merely by adding an “s,” but the wordsmay be considered to be have sufficient similarity of use in that theyare both capable of being pluralized.

In an example, for a fill-in sentence question, for each potential wronganswer choice (e.g., words other than the correct word which have notalready been weeded out based on other tested conditions), the systemand method searches the indices database for all sentences associatedwith the potential wrong answer choice for a sentence in which thepotential wrong answer choice includes a similar suffix (and/or) prefixas that of the correct word. If the search returns no results, thesystem and method considers the word as not being able to accept asimilar suffix (and/or) prefix as that of the correct word, and removesit from the corpus of possible wrong answer choices for the fill-in theblank questions.

In an example embodiment of the present invention, the system and methodmay provide a designated word in step 230 and, in a synonym questionmode, may output a set of multiple choice answers from which the usermay select one as a synonym. The system and method may include only onepossible synonym, as one of the choices, and may automatically select apredetermined number of wrong answers for inclusion as the otherchoices, where the predetermined number of wrong answers may include,for example, a number of antonyms.

In an example embodiment, the system and method may use the lexicaldatabase(s) in step 260 to query a corpus of words to determineword/sense pairs that are synonyms or close relations of the designatedword/sense pair. In the example embodiment, only a single word/sensepair indicated by the database(s) as a direct synonym may be provided asone of the multiple choices in step 261. A determination of a synonymmay depend on such factors as the POS of the designated word, similarityof meaning to the designated word, similarity of use to the designatedword, similarity of frequency of use to the designated word, and/orskill level.

For example, the system and method may analyze the corpus of words withrespect to a combination of the above-enumerated factors. For example, adesignated word may be the word “content.” In this example, “content”may refer to a feeling of satisfaction or happiness and the POS of“content” may be an adjective. An appropriate determination by thelexical database(s) may be that the word “satisfied” is a direct synonymof the word. In this context, the word “information” may not be asynonym to “content.” The word “satisfied” may have a use with similarcharacteristics as those of “content,” a closest frequency of use, and apredetermined threshold skill level.

The system and method may exclude all of the remaining results of thequery indicated to be a synonym or having a very close meaning to thatof the designated word/sense pair from the set of wrong answer choicesin step 262, i.e., to remove the possibility of multiple correctanswers. Word/sense pairs that may not be a synonym but may be catalogedas similar to or “see also” may be also be removed from possible wronganswers. Additionally, word/sense pairs that may otherwise be considereda synonym using a different POS or different meaning may also be removedfrom possible answers. For example, where “satisfied” is provided as apossible answer choice to “content,” the word “information” may beremoved from being a candidate for being provided as a possible choice.

In an example embodiment, the lexical database(s) may use a series ofpointers from the designated word branching out a number of levels fromword/sense pair to word/sense pair. The series of pointers may connectthe designated word with direct synonyms. The system and method mayselect one of the word/sense pairs pointed to from the designated wordas a synonym. For pointers to other word/sense pairs that are directsynonyms, or synonyms if an alternative definition or POS of theword/sense pair was used, the system may traverse such pointers in step262 and omit then from the multiple choice answers. For example, wherethe word/sense pair may be “content” and “satisfied” is determined bythe lexical database(s) to be a direct synonym, pointers to word/sensepairs corresponding to “substance” or “matter” may be traversed andomitted.

In an example embodiment, with respect to similarity of use, the systemand method may apply different conditions for inclusion of a word/sensepair as one of the wrong answer choices for a synonym question.

For example, if the designated word for the synonym question is taggedas a noun, the system and method may further require that both thedirect synonym and all wrong answer choices share the same set of uniquebeginners as that of the designate word, which is a noun. If thedesignated word is tagged as a verb, the system and method may requirethat both the direct synonym and all wrong answer choices share the sameset of verb frames as that of the designated word. If the designatedword is tagged as an adjective, the system and method may require thatboth the direct synonym and wrong answer choices share the attributionalproperty as that of the designated word.

In an example embodiment, where a direct synonym is presented as apossible answer choice, the system and method may be configured toensure that another synonym for an alternative definition is not alsoprovided as an answer choice. In an example embodiment where adesignated word may have multiple definitions (such as the “content”example above), the system and method may be configured to ensure thatan item that corresponds to a synonym of a separate definition orseparate POS is not provided as a wrong answer choice.

In an example embodiment, where a synonym question is asked, the systemand method may determine direct antonyms to the designated word andprovide these antonyms as wrong answer choices in step 263. The systemand method may use a lexical database(s) to query a corpus of words todetermine word/sense pairs that are antonyms of the designated word, andmay provide one or more of these antonyms as wrong answer choices. In analternative example embodiment, words may be selected which are neithersynonyms nor antonyms of the subject word. In an example embodiment,both antonyms and words that are neither synonyms nor antonyms may beselected as the wrongs answers. If a user does not answer the questioncorrectly in step 264, the user may encounter the question again in step265.

In an example embodiment of the present invention, in an antonymquestion mode, the system and method may provide a designated word instep 230 and may output a set of multiple choice answers from which theuser may select one as an antonym. The system and method may includeonly one possible antonym, as one of the choices, and may automaticallyselect a predetermined number of wrong answers for inclusion as theother choices, where the predetermined number of wrong answers mayinclude a number of synonyms. Alternatively, words that are neithersynonyms nor antonyms may be selected as the wrong answers. In anexample embodiment, both synonyms and words that are neither synonymsnor antonyms may be selected as the wrongs answers.

In an example embodiment, the system and method may use the lexicaldatabase(s) in step 250 to query a corpus of words to determineword/sense pairs that are antonyms of the designated word or areantonyms of word/sense pairs that have close relations of the designatedword. In the example embodiment, only a single word/sense pair indicatedby the database(s) as an antonym may be provided as one of the multiplechoices in step 251. A determination of an antonym may depend on suchfactors as the POS of the original word, a definition that may beopposite that of the meaning to the designated word, similarity offrequency of use to the designated word, and/or skill level.

For example, the system and method may analyze the corpus of words withrespect to a combination of the above-enumerated factors. For example, adesignated word may be the word “down.” In this example, “down” mayrefer to a feeling of unrest or depression. An appropriate determinationby the lexical database(s) may be that the word “cheerful” is an antonymof the word. In this context, the word “up” describing a direction maynot an antonym to “down.” The word “cheerful” may have a use withsimilar characteristics as those of “down,” a closest frequency of use,and a predetermined threshold skill level.

The system and method may exclude all of the remaining results of thequery indicated to be an antonym of the designated word or being anantonym of a word having a very close meaning to that of the designatedword/sense pair from the set of wrong answer choices, i.e., to removethe possibility of multiple correct answers in step 252. Additionally,word/sense pairs that may otherwise be considered an antonym using adifferent POS or different meaning may also be removed from possibleanswers. For example, where “cheerful” is provided as a possible answerchoice to “down,” the word “up” may not be provided as a possiblechoice. In an example embodiment, word/sense pairs that may not be anantonym but may be cataloged as dissimilar to the subject word may bealso be removed from possible wrong answers in step 252. Alternatively,dissimilar words that are not antonyms may be included as wrong answerchoices.

The system and method may also remove antonyms that may contain the rootof the designated word. Examples of this may include words that mirrorthat designated word, but may have a prefix attached. For example, ifthe designated word is “satisfied,” the word “dissatisfied” may not beprovided as a possible antonym answer choice because it contains thesame root as “satisfied” and would be an obvious answer choice.

In an embodiment, the lexical database(s) may use a series of pointersfrom the designated word branching out a number of levels fromword/sense pair to word/sense pair. The series of pointers may connectthe designated word with direct antonyms. The system and method mayselect one of the word/sense pairs pointed to from the designated wordas an antonym. Pointers to other word/sense pairs that may be directantonyms, or antonyms if an alternative definition or POS of theword/sense pair is used, may be traversed for omission from appearing asanswer choices. For example, where the word/sense pair may be “down” and“cheerful” is determined by the lexical database(s) to be an antonym,pointers to word/sense pairs corresponding to “up” or “above” may betraversed for removal of candidates for answer choices.

In an example embodiment, with respect to similarity of use, the systemand method may apply different conditions for inclusion of a word/sensepair as one of the wrong answer choices for an antonym question.

For example, if the designated word is tagged as a noun, the system andmethod may further require that both the direct antonym and all wronganswer choices share the same set of unique beginners as that of thedesignated word, which is a noun. If the designated word is tagged as averb, the system and method may require that that both the directantonym and all wrong answer choices share the same set of verb framesas that of the designated word. If the designated word is tagged as anadjective, the system and method may require that that both the directantonym and wrong answer choices share the attributional property asthat of the designated word.

In an example embodiment, where an antonym is presented as a possibleanswer choice, an antonym to an alternative definition may not also beprovided as an answer choice. In an example embodiment, where adesignated word may have multiple definitions (such as the “down”example above), a choice that may correspond to an antonym of a separatedefinition or separate POS may not be provided as a wrong answer choice.

In an example embodiment where an antonym question is asked, the systemand method may determine synonyms to the designated word and providethese synonyms as wrong answer choices in step 253. The system andmethod may use a lexical database(s) to query a corpus of words todetermine word/sense pairs that are synonyms of the designated word, andmay provide one or more of these synonyms as wrong answer choices instep 253. If a user does not answer the question correctly in step 254,the user may encounter the question again in step 255.

In an example embodiment of the present invention, in a definitionquestion mode, the system and method may highlight a designated word instep 230 and may output a set of multiple choice answers from which theuser may select a definition of the designated word. The system andmethod may include only one possible definition, as one of the choices,and may automatically select a predetermined number of wrong answers forinclusion as the other choices, where the predetermined number of wronganswers may include definitions of dissimilar words or antonyms.

In step 270, the system and method may use the electronic dictionary todetermine a correct definition of a designated word. A definition asprovided by the dictionary may depend on the POS of the designated word.

The system and method may use a lexical database(s) in step 271 to querya corpus of words to determine meanings that may be closely related tothe correct definition. The lexical database(s) may also determine thedefinition of word/sense pairs that are synonyms or close relations ofthe designated word. In the example embodiment, only the definitiongiven by the electronic dictionary may be provided as one of themultiple choices in step 272. Definitions of synonyms or meanings thatmay closely match the correct definition may be discarded as answerchoices.

The system and method may exclude all of the remaining results of thequery indicated to be a synonym of the designated word or having a veryclose meaning to the correct definition in step 273. Definitions ofword/sense pairs that may not be a synonym but provide a definition thatmay be similar to the correct definition may also be removed frompossible wrong answers. Additionally, definitions of word/sense pairsfor the designated word using a different POS or different meaning mayalso be removed from possible answers. For example, where “a feeling ofsatisfaction” is determined by the dictionary as the definition of“content,” and therefore provided as a possible answer choice, thedefinition “of or relating to the substance of a matter” may not beprovided as a possible choice, even though this may be a definition of“content.”

In an example embodiment, the lexical database(s) may use a series ofpointers from the designated word branching out a number of levels fromword/sense pair to word/sense pair. The series of pointers may connectthe designated word with various definitions that may be similar to thedefinition of the designated word. The system and method may select oneof the word/sense pairs pointed to from the designated word as a correctdefinition. Pointers to other similar definitions, or definitions ofsynonyms of the designated word if an alternative definition or POS ofthe word/sense pair is used, may be traversed and removed from theanswer choices. For example, where the word/sense pair may be “content”and “a feeling of satisfaction” is determined by the dictionary to bethe correct definition, pointers to definitions related to othermeanings of “content,” i.e., “of or relating to the substance of amatter,” may be traversed.

In an example embodiment, the system and method may apply differentconditions for inclusion of a definition as one of the wrong answerchoices in step 274. For example, if the designated word is tagged as anoun, the system and method may further require that all wrong answerchoices contain definitions of word/sense pairs that correspond to anoun. If the designated word is tagged as a verb, the system and methodmay require that all wrong answer choices contain definitions ofword/sense pairs that correspond to a noun. If the designated word istagged as an adjective, the system and method may require that wronganswer choices contain definitions of word/sense pairs that correspondto a noun.

In an example embodiment, where the correct definition is presented as apossible answer choice, a similar definition may not also be provided asan answer choice in step 274. In an example embodiment, where adesignated word may have multiple definitions (such as the “content”example above), a choice that may correspond to a definition similar tothat of a different meaning of the designated word may not be providedas a wrong answer choice.

In an example embodiment, where a definition question is asked, thesystem and method may use a lexical database(s) to determine definitionsof antonyms to the designated word and provide these definitions aswrong answer choices. The system and method may use any definitions forwrong answer choices, except for the definitions which have beenpreviously discarded. If a user does not answer the question correctlyin step 275, the user may encounter the question again in step 276.

The system and method may also generate hint questions for the user toanswer. A hint question may ask a user a question about a designatedword, i.e., to select a synonym, but, in addition to being provided witha list of answer choices, a user may also be provided with a sentencethat uses the designated word, which may serve as a “hint” to the user.A user may then use the provided sentence to assist in the user'sdetermination of the synonym of the designated word.

The system and method may provide two types of hint questions: synonymhint questions and definition hint questions. In an example embodimentof the present invention, the system and method may ask a user todetermine a synonym of a designated word and may provide an examplesentence containing the designated word from the index of sentences, asa hint, to illustrate the use of the designated word. In another exampleembodiment, the system and method may ask a user to determine adefinition of a designated word and may provide a supplemental sentencecontaining the designated word from the index of sentences as a hint.The system may provide example sentences for both synonym hint anddefinition hint questions which contain the designated word where it isused in a way consistent with the target word/sense pair, but thedefinition is not readily apparent.

In an example embodiment, the system and method may provide the userwith a synonym hint question by highlighting a designated word within aprovided sentence and outputting a set of multiple choice answers fromwhich the user may select one as a synonym. The system and method mayinclude only one possible synonym, as one of the choices, and mayautomatically select a predetermined number of wrong answers forinclusion as the other choices, where the predetermined number of wronganswers may include, for example, a number of antonyms. The user may usethe provided sentence to assist in a determination of a synonym.

The right and wrong answer choices for a synonym hint question may begenerated using the same method for generating right and wrong answerchoices for synonym questions, i.e. querying a lexical database todetermine synonyms, selecting a synonym, and removing other synonymsfrom being wrong answer choices.

In an example embodiment of the present invention, the system and methodmay provide the user with a definition hint question by highlighting adesignated word within a provided sentence and outputting a set ofmultiple choice answers from which the user may select a definition ofthe designated word, in the context of the sentence. The system andmethod may include only one possible definition, as one of the answerchoices, and may automatically select a predetermined number of wronganswers for inclusion as the answer other choices, where thepredetermined number of wrong answers may include definitions ofdissimilar words or antonyms. The user may use the provided sentence toassist in a determination of a definition.

The right and wrong answer choices for a definition hint question may begenerated using the same method for generating right and wrong answerchoices for definition questions, i.e., querying a lexical database todetermine actual definitions of the generated words, selecting a correctdefinition, removing other similar definitions from being wrong answerchoices, and providing definitions of other words as possible wronganswer choices.

In an example embodiment, with respect to skill level, the system andmethod may limit the wrong answer choices to those that have beenindicated to be likely to appear in a standardized test, such as the SATor GRE.

For all question types, the system and method may analyze a large corpusof text (e.g., including more than one billion words) to determine arespective frequency of each word of the electronic dictionary, and sortthe words by their respective frequencies. Of all of the possible wronganswer choices that satisfy all of the other applied conditions, thesystem and method may select those whose respective frequency values areof the (e.g., three in an instance where three wrong answer choices areprovided) shortest absolute distances to the frequency value of thecorrect word. For example, if the correct word appears 345 times, theselected wrong answer choices might have respective frequency values of340, 347, and 348, frequency values of all other possible words beingeither less than 340 or greater than 350, which are of greater absolutedistance to the frequency value of the correct answer than the greatestabsolute distance of 5 of the frequency values of the selected wronganswer choices from the frequency value of the correct answer.

In an example embodiment, presented sentences (for fill-in sentences,synonym hint sentences, and definition hint sentences) or designatedwords may be ranked by difficulty and by ability to discriminate betweendifferent skill levels of test takers, and may be selected based onthose rankings. In an example embodiment of the present invention, asentence's difficulty and discrimination scores may be calculated in twostages, where, in the first stage, the scores are calculated prior tobeing output as a test question, and, in the second stage, the scoresare recalculated each time an answer selection is made.

In the first stage, a sentence's difficulty score may be based solely onthe frequency value (described above) of the correct answer, where thegreater frequency values represent lesser sentence or designated worddifficulty. The sentence frequency values may be further grouped intodifficulty categories, e.g., ranging from −10 (easiest) to 10 (hardest).Students' abilities may be similarly grouped into ability categories,e.g., ranging from −10 (least ability) to 10 (greatest ability). For aparticular test taker, sentences are selected from those having theclosest difficulty category matching the ability category of the testtaker, e.g., a sentence or designated word having a difficulty of −6 fora test taker having an ability of −6 (or −7 where there are no sentenceshaving a difficulty of −6, e.g., to which the test taker has not alreadyresponded or, alternatively, that have not already been output to thetest taker). In an example embodiment, the system and method may excludequestions that have already been answered by the test taker or,alternatively, that have already been output to the test taker.

Prior to answering any questions, the system and method may initiallyassign the test taker to a predetermined ability category. Subsequently,as the test taker answers questions of varying difficulty, the systemand method may, e.g., continuously, reassign the test taker to differentability categories. For example, the system and method may assign thetest taker to the ability category corresponding to a difficultycategory of whose questions the test taker answers 50% correctly. In anexample embodiment, the ranking of the test taker may be based on acombination of the test taker's performance during a present testsession and prior test sessions. Accordingly, the system may maintain arecord of a test taker's performance from session to session.

Moreover, of those sentences having a matching difficulty rank, thosesentences which have the highest discrimination scores are selected foroutput. Discrimination scores may range, e.g., from 0 to 2. In anexample embodiment, all sentences for which answers have not yet beeninput may be initially set to the same predetermined discriminationscore, e.g., a low score, which may be changed in the second stage asdescribed below.

In the second stage, the system and method may determine a newdifficulty category and a new discrimination category for a questionafter, and based on, each response to the question. For example, thesystem and method may divide all the test takers who have answered thequestion into a series of ability groups corresponding to the abilitycategories by which each of the respondents have been ranked, e.g.,ranging from −10 to 10, and calculate, for each group, the percentage ofrespondents of the group who correctly answered the question. The systemand method may then plot the ability groups against the respectivepercentages calculated for those ability groups, and determine the newdifficulty and discrimination categories to which the question isassigned based on the plotted data.

For example, Table 1 below represents an instance where 199 users haveanswered a particular question, and indicates, for each representedability group, the number of users of the group who have answered thequestion, the number of those who have answered the question correctly,and the calculated percentage. FIG. 4 shows a graph, corresponding toTable 1, in which the scale of ability groups ranging from −10 to 10 isrepresented by the abscissa, and the scale of calculated percentagesranging from 0% to 100% is represented by the ordinate.

TABLE 1 Ability Groups ability group users correct % 10.0 12 12 100 8.08 7 87.5 6.0 7 7 100 4.0 13 12 92.31 2.0 13 13 100 0.0 13 13 100 −2.0 9552 54.74 −4.0 13 0 0 −6.0 16 4 25 −8.0 7 1 14.29 −10.0 2 0 0 Total: 199121 60.8

For determining the categories based on the plotted values, the systemand method applies a curve fitting algorithm to determine a best fitcurve for the plotted values, for example, as shown in FIG. 4. Theplotted points may be weighted differently depending on the number ofrespondents of the plotted ability group. For example, while the circlesin FIG. 4 might not be drawn to scale, their different sizes areintended to correspond to the respective numbers of respondents of therespective represented ability groups. For example, the largest numberof respondents were of ability group −2.0, around which point thelargest circle is drawn. The greatest weight may be given to the pointplotted for group −2.0 because the greatest number of respondents are ofthat group.

The system and method then assigns the ability category closest to theplotted point at which the calculated curve crosses the x-axis(corresponding to 50% on the ordinate scale) as the difficulty categoryof the question. The system and method also then assigns the slope ofthe calculated curve at the plotted point at which the calculated curvecrosses the x-axis as the discrimination score of the question.

In an alternative example embodiment of the present invention, theability category closest to the plotted point at which the calculatedcurve crosses the x-axis is used as one of the input parameters to anequation for calculating the difficulty category. For example, thedifficulty category may continue to be based partially on the frequencyvalue (described above) of the correct answer of the question. In avariant of this alternative, each time the question is answered and thedifficulty category recalculated, the frequency value may be weightedless in the calculation of the difficulty category, e.g., until itsweight is zero.

In an example embodiment of the present invention, in an instance wherea test taker's ability category changes during a test taking session,the system and method recalculates the difficulty and discriminationcategories for any question that was previously answered by the testtaker during the same test taking session and prior to the re-ranking ofthe test taker's ability, and for which the difficulty anddiscrimination categories were previously calculated during the sametest taking session, after the test taker answered the question, andprior to the re-ranking of the test taker's ability.

With the re-categorization of the question, the test taker's rankingmight not be accurately reflected. Accordingly, in an example embodimentof the present invention, after re-categorization of the question, thesystem and method may re-perform the calculation for ranking the testtaker. If the test taker is thereby re-ranked, the system and method maycycle back to re-categorize the questions. The system and method maycontinue to cycle in this way until the categorization of the questionsand the ranking of the test taker stabilize.

In an example embodiment of the present invention, the system and methodconditions an initial categorization of questions based on a testtaker's performance during a particular session on the test taker havinganswered a predetermined number of questions during the particularsession, thereby increasing the probability that the initialcategorization of the questions is based on an accurate ranking of thetest taker's ability during the particular session.

In an example embodiment, the system and method may refrain fromre-calculating the difficulty and discrimination categories of thosequestions whose categories were calculated based on questions answeredby the re-ranked test taker in a prior session of the test taker and notthe current session of the test taker, because it may be assumed thatthe re-ranking of the test taker across multiple sessions represents anactual change in the test taker's ability over time, whereas are-ranking within a single session may be considered to reflect a changein data during the session regarding the test taker's ability whichability remains stable throughout the single session.

The system and method of the present invention may allow for an adaptivelearning system which may tailor questions and explanations of thecorrect answer to the presented questions to the test taker, in a mannerthat may allow for the test taker to improve the test taker's vocabularyacquisition and retention. FIG. 5 is a screen shot of an exampleinterface of the adaptive learning system 20. A test taker may be askedone question at a time which question may be displayed on the interfaceas depicted in FIG. 5. A test taker may be asked a number of questionsin a series of rounds, to allow for an optimization of learning for thetest taker. The number of questions per round may be selectively chosen,and in an example embodiment, may consist of 10 questions per round. Thequestions presented to a test taker may be separated into fourcategories of question: assessment, review, progress, and masteryreview.

An assessment question is a question that is directed towards brand newmaterial that the user has not previously encountered, allowing thesystem to determine a user's ability. An assessment question may, forexample, involve a new designated word that has not been previouslypresented to the user. The system may store and update for a user arespective Active Learning List which may catalog all of the words thatthe user has encountered and is working towards learning. The system maybe configured to provide assessment questions such that words already ona user's Active Learning List are not included in assessment questionsoutput to the user. Once a user answers an assessment question, thedesignated word in the assessment question may be added to the ActiveLearning List. The Active Learning List may indicate whether the usercorrectly answered a question about the designated word, and is makingprogress on learning the word, or incorrectly answered the questionabout the designated word.

A review question is an exact question, verbatim, that the userpreviously answered incorrectly in a previous round. When a question isincorrectly answered by a user, it is placed on the user's ActiveLearning List, and the list may indicate that the user has incorrectlyanswered a question concerning the word. The system may be configured toprovide review questions such that only words that are on a user'sActive Learning List, that were answered incorrectly, are tested by thereview questions. The system may be configured such that a reviewquestion is repeated after (at least) a predefined number of questionsare output after the question, testing the same word tested by thereview question, was incorrectly answered by a user.

A progress question is a question that is presented as a follow-upquestion concerning a designated word that was the subject of apreviously output question a user correctly answered. By testing a useragain about a designated word, these progress questions may demonstratea user's comprehension of the word. Since progress questions are onlyasked about words the user has already encountered, according to anexample embodiment, progress questions only test words from a user'sActive Learning List, for example after a review question testing theword has been correctly answered or after the user correctly answers anassessment question or another progress question about the designatedword. If subsequent progress questions are answered correctly about adesignated word, a word may be marked as “mastered” in the ActiveLearning List.

A mastery review question is a question about a “mastered” word in theActive Learning List. Mastery review questions appear less frequentlythan progress or review questions and are designed to ensure that a userstill remembers that definition, synonym, or antonym of a designatedword.

Questions that are categorized as assessment questions, may be presentedto the user based on the user's ability (described above). In an exampleembodiment, a user who correctly answers an assessment question about adesignated word, may subsequently receive a progress question in a laterround, about the designated word. The system and method may note thatthe assessment question was answered correctly and may discard thequestion from being asked in a subsequent round. If the user does notanswer an assessment question correctly, the system and method may notethat the question was answered incorrectly, and may output the questionin a subsequent round as a review question. If a question is answeredincorrectly, the designated word may be added to a generated list ofwords that the user may be learning (Active Learning List).

In an example embodiment, where a user incorrectly answers any type ofquestion in steps 223, 254, 264, and 274, the system and method mayre-present the same question as a review question in a subsequent round.If a user correctly answers a review question, the system and method maynote that the review question was answered correctly and may discard thequestion from being asked again. If the user does not correctly answer areview question, the system and method may make a notation that thequestion was answered incorrectly, and may output the question again ina subsequent round.

In an example embodiment, like review questions, progress questions mayalso test words from the Active Learning List, but do not includequestions that were already presented to the user. A progress questionmay be a follow up question about a designated word for which the usercorrectly answered a question. A progress question may test on the samedesignated word, but may test a different meaning of the word, or maypresent a new question and/or question type testing the same meaning.

In an example embodiment of the present invention, a progress questionmay be provided for a designated word after a user correctly answers aprevious assessment, review, or even a previously provided progressquestion concerning a word on the Active Learning List. A progressquestion may be a follow-up question about the designated word which maytest a user on additional definitions or uses. If a user correctlyanswers a progress question, the system and method may mark anyprogression on the word in the Active Learning List in step 225,including the number of questions that the user has answered correctlyabout the designated word. For example, the system may indicate thenumber of questions concerning the word the user, e.g. consecutively,correctly answered. The system and method may output additional progressquestions testing the same designated word. If a user incorrectlyanswers a subsequent progress question, the user may be presented withthe incorrectly answered question as a review question in a subsequentround in steps 224, 255, 265, or 276.

In an example embodiment, where an individual correctly answers apredetermined number of progress questions about a designated word in arow, the word may be classified as mastered in step 280. In step 290 thedesignated word may be marked as mastered on the Active Learning List.The number of questions that a user must answer correctly to achievemastery on a designated word may be based on the number of questions inthe index for that word. A user that has mastered a designated word instep 280, may encounter questions concerning the mastered word at areduced frequency in subsequent rounds.

In an example embodiment of the present invention, a particulardefinition for a designated word may be emphasized for the user forcomprehension. This may occur, for example, in an instance where theuser is presented with a sentence using the designated word in a contextconsistent with the particular definition, or the presented synonyms orantonyms relate to the particular definition of the designated word. Inthis embodiment, an administrator, i.e., a teacher or someone in aneducation setting, may manually tag words in a corpus of sentences toindicate whether its definition as used in a respective sentence is akey definition. Alternatively, the words of the corpus of sentences maybe tagged with their definitions in the given context, and the systemmay treat the definition that most often is tagged for the given word asthe key definition. Progress questions presented to the user may beconcentrated on testing the user on the particular definition for thedesignated word.

Questions that may be categorized as mastery review questions, may testwords already designated as a mastered word in the Active Learning List.Unlike regular review questions, mastery review questions presentquestions to a user that the user has not seen. These mastery reviewquestions may be intended to reinforce that a user understands thedefinition and use of a particular word. If a user correctly answers amastery review question, the presented mastery review question may bediscarded and the user may receive a subsequent mastery review questionin a later round. If a user incorrectly answers a mastery reviewquestion, the designated word may be removed from the list of masteredwords. That is, the designated word may no longer be marked on theActive Learning List as mastered, and the user is no longer consideredto have mastered the word. If a mastery review question is incorrectlyanswered, the missed question may be presented again to the user, inidentical form, as a review question in a subsequent round. The user mayagain master the designated word by correctly answering the reviewquestion (which is simply the missed mastery review question), and againanswering a series of progress questions about the designated word. If auser incorrectly answers a mastery review question about a designatedword, the system may be configured such that it does not again indicatethe user to have mastered the word in the same session. Re-mastery ofthe designated word may be made in subsequent sessions. For example, thesystem may require the user to log-out and re-log-in in before updatingthe user history to indicate that the user has mastered the word.

The system and method of the present invention may operate in twodistinct modes when a user is not logged in: an experimental mode whichmay tailor the adaptive learning system to output questions that maybenefit all users in the system, and a non-experimental mode in whichthe questions and explanations of the correct answer are tailored tobenefit an individual user. For example, the system may operateaccording to the experimental mode when a user makes use of the systemas a guest, without payment of a fee, and according to thenon-experimental mode when a fee is paid.

In an example embodiment of the present invention, the system mayoperate in an experimental mode. In this mode, the system may outputquestions tailored for compiling information about the output questions,based on responses to the output questions, to determine theirsuitability (as described above) to all users of the system. A number ofquestions for designated words may have been output to users with lessfrequency than other questions, thus limiting information to the systemabout the suitability of these output questions. The system maytherefore output such questions more frequently for users in theexperimental mode to determine their suitability for general usage.

During experimental mode, a user may operate as an anonymous user, wherethe user's progress is not tracked by the system from session tosession. As the user is not logged in, the user does not have access tothe user's Active Learning List. Therefore, the system may be configuredsuch that the questions presented to the user are not based on theuser's Active Learning List. During experimental mode, a user may bepresented with infrequently output questions about a designated word inan effort to allow the system to obtain valuable information about thenature and suitability of the presented question. These questions may beorganized according to their playcount, namely how often the questionhas been output to users. An experimental question with the lowestplaycount may be selectively presented to the anonymous user, forexample, conditional upon that the question's difficulty is within therange of +1 to −1 from a determined user ability of the anonymous user.As discussed above, where a user's ability has not yet been fullydetermined, an initial ability categorization may be made. For example,when an anonymous user begins a session, the system may initially assignthe anonymous user to a predetermined ability category, and may output aplurality of questions having varying difficulty to determine the user'sability category.

In an example embodiment of the present invention, the system maypresent an experimental question with the lowest playcount to ananonymous user conditional upon that the anonymous user has not beentested by another question about the designated word in the experimentalquestion, e.g., in the same session. An experimental question may alsobe presented to the user conditional upon that the question has not beenpresented to the user within the last 14 days, e.g., where the user isnot anonymous but has not logged in with certain privileges allowing theuser to use the system in a non-experimental mode.

In an example embodiment of the present invention, the system may beoperating in a non-experimental mode. As with the experimental mode, thesystem may operate in the non-experimental mode when a user is notlogged in. In the non-experimental mode, the response to the outputquestions may generate subsequent questions that are most suitable forthe individual anonymous user. Therefore, the non-experimental modediffers from the experimental mode in that the non-experimental modemodels suitability for an individual user rather than all of the systemusers, even though a user is not logged in. The presented questions maybe tailored to the anonymous user, where the anonymous user may bepresented with suitable questions in accordance with the discriminationlevel of the question, and the questions may be organized according totheir discrimination level. A question with the highest discriminationmay be selectively presented to the anonymous user if the question'sdifficulty is within the range of +1 to −1 from a determined userability for the anonymous user. As discussed above, where an anonymoususer's ability may not been fully determined, an initial abilitycategorization may be made.

In an example embodiment of the present invention, the system maypresent a suitable question with the highest discrimination to ananonymous user conditional upon that the anonymous user has not beentested by another question about the designated word in the presentedquestion, in the same session. A question may also be presented to theuser conditional upon that the question has not been presented to theuser within the last 14 days.

In an example embodiment of the present invention, the system mayoperate when a user is logged into the user's account (e.g., as amember). When a user is logged in, the system may operate in yet anothermode, the logged-in mode, different than the experimental andnon-experimental modes. The system may allow the logged in user to haveaccess to the user's Active Learning List and may cause the system togenerate questions based on the Active Learning List that are mostsuitable to the user, which further allows the adaptive learning systemto be tailored to the user.

Questions may be presented to the logged in user in rounds of 10questions, and each question in the round may be presented to the userindividually as shown in FIG. 5. Initial questions may be presented tothe user based on a determined ability of the user. If a user ispresented with a question that tests on a designated word that the userhas not encountered, the designated word may also be added to the ActiveLearning List. This entry to the Active Learning List may denote thatthe user correctly answered a question about the designated word, and ismaking progress on the designated word.

As discussed above, a user may be first given an assessment question ona new designated word to test the user's ability. Responsive to aselection of the correct answer, the system may record progress of thedesignated word in the Active Learning List in step 225. A user may thenbe presented with a series of subsequent progress questions which maytest the user on alternative definitions or uses of the designated word.If a user correctly answers the subsequent progress questions correctly,the word may be considered mastered in step 280, and marked accordinglyin the Active Learning List. If a word is deemed mastered by a user, theuser may only encounter questions about the designated word duringmastery review questions, which may occur much more infrequently.

In an example embodiment, if any of the assessment, progress, or masteryreview questions are answered incorrectly, the user may encounter thesame question again in a subsequent round, as a review question. Noprogress may be recorded in the Active Learning List if an incorrectanswer was selected and progress may not be made on a designated word ifit was incorrectly answered in the same session. If a user answers areview question correctly (in a subsequent session), the user may bepresented with subsequent progress questions.

Assessment questions, which may be presented to the user to test newwords, may be organized in accordance with their discrimination leveland presented to a user based on the discrimination level of thequestion. In an example embodiment, an assessment question with thehighest discrimination may be selectively presented to the user if theassessment question's difficulty is with the range of +0.5 to −0.5 froma determined user ability. As discussed above, where a user's abilitymay not been fully determined, an initial ability categorization may bemade.

In an example embodiment of the present invention, the system maypresent a suitable assessment question with the highest discriminationto the user conditional upon the user not having been tested by anotherquestion on the same designated word. An assessment question also maynot contain a designated word on a user's Active Learning List and maynot be a designated word that the user is working on.

If an assessment question is answered incorrectly by a user thedesignated word may be placed in the Active Learning List. The user maybe presented with the question as a review question in a subsequentround, such as steps 224, 255, 265, and 276. As previously discussed,progress may not be made on a designated word if the assessment questionwas answered incorrectly in the same session. If an assessment questionis answered correctly, the user may be presented with subsequentprogress questions.

Progress questions are on words in a user's Active Learning List and mayrepresent questions that are presented to the user to test alternativedefinitions or uses of a designated word after the user had correctlyanswered a question about the designated word in a previous round.Questions that test on a new word that was not previously presented tothe user, may not be presented as progress questions.

Progress questions may not be presented on words that were previouslyanswered incorrectly within the same session. A question that has beenpreviously presented to the user may not be used as a progress question(a correctly answered question may be discarded and an incorrectlyanswered question may be presented as a review question).

In an example embodiment, where the user incorrectly answered a questionabout the designated word, the user may be presented with theincorrectly answered question as a review question, and must wait untila new session before being presented with a progress question about thedesignated word if the user answers the review question correctly. Uponcorrectly answering the review question, in a new session, a questionmay be presented as a progress question for the designated word wherethe question tests the same sense as the previously incorrectly answeredquestion, and conditional upon that the progress question is within therange of +3 to −10 from a determined user ability. In an exampleembodiment, the system may present progress questions in accordance withthe discrimination level of the question, with questions with thehighest discrimination being asked first.

In an example embodiment of the present invention, where no possibleprogress questions exist that are within the range of +3 to −10 from adetermined user ability, questions outside this range may be presentedas progress questions, with questions being prioritized according toproximity to a user's ability. In an example embodiment where nopossible questions exist that are consistent with the sense of thedesignated word, questions concerning the designated word for adifferent sense that has not been presented to the user may be presentedas progress questions, unless these questions have been previouslypresented to the user.

Review questions may encompass questions that were previously presentedto the user as either an assessment question or a progress question thatthe user answered incorrectly. Review questions may be continuallypresented to the user in the review mode, in subsequent rounds, untilthe question is answered correctly. If a review question is answeredcorrectly by a user, the question may be removed from future rounds, andthe user may be presented with a progress question about the designatedword, in a future session. According to an example embodiment whereprogress may not be made within the same session on previouslyincorrectly answered questions of designated words, a progress questionmay not be asked about a designated word that was in a review question,until another session.

Mastery review questions may represent questions selected for words thathave already been mastered by a user. Words may be mastered by a user bycorrectly answering the assessment and progress questions of adesignated word. Words may still be mastered even if a user incorrectlyanswers a question about the designated word in an earlier session, ifthe user answers subsequent review and progress questions about thedesignated word correctly in later sessions. If a user answered everyquestion pertaining to a designated word correctly during the user'sprogression, the user may not encounter sentences with the designatedword in a mastery review mode. Words that may be marked as mastered inthe previous week, may not be eligible for mastery review.

As described above, questions may be presented to the user in rounds of10 questions. FIG. 6 is a flowchart that illustrates a process ofdetermining the allocation of the questions by question category in eachround. An allocation of each of the 10 slots in each of the rounds maybe made in step 300 to allow for the user to be presented with acombination of assessment questions, review questions, progressquestions, and mastery questions. In an embodiment of the presentinvention, the ordering of the 10 questions may be arbitrary.

In an example embodiment, the allocation of the 10 slots for questionsfor both experimental and non-experimental modes may be made accordingto a predetermined distribution. Of the 10 slots, up to two slots may beallocated for review questions repeating questions that the userincorrectly answered as noted in step 304. If a user previously answeredless than two questions incorrectly, any outstanding review questionslots may be allocated to assessment questions. For example, if a userhad previously incorrectly answered only one question, or only had oneincorrect question outstanding from previous rounds that was neverpresented as a review question, the system and method may be configuredto limit to only one of the 10 slots allocation of a review question.The leftover review question slot may be allocated instead to assessmentquestions.

If a user previously answered more than two questions incorrectly, thesystem and method may be configured to, nevertheless, allocate only twoof the 10 slots to review questions, and the outstanding reviewquestions may be carried over for use in a subsequent round. Forexample, if a user had previously incorrectly answered four questions inprevious rounds, only two slots of the 10 slots may be allocated torepresenting review questions. The outstanding two incorrectly answeredquestions may be allocated as review questions in a subsequent round.

After up to two of the 10 slots are allocated to review questions instep 304, the remaining eight slots may be allocated randomly using aprobability distribution. In step 306, 90% of the eight slots may beallocated to progress questions (seven slots), and 10% of the remainingslots may be allocated to mastery review questions (one slot).

Assessment questions may be allocated to any remainder slots in steps318 and 320 after the allocation of the review, progress, and masteryreview questions. In an example embodiment of the present invention,where there are no review questions in step 302, i.e., the user has notincorrectly answered a previous question, the user may be presented withseven progress questions and one mastery review question in step 306.The remaining two slots may be allocated to assessment questionpresenting new words.

In an example embodiment, where the system runs out of mastery questionsin step 310, this slot may be filled with a progress question in step312. The system may allocate the remaining slot to a progress questionunless the system has run out of progress questions in step 308, andtherefore an assessment question may be allocated in step 318. In anexample embodiment, where the system runs out of progress questions, acheck for mastery review questions may be made in step 314, and ifpresent, additional master review questions may be added in step 316. Ifno mastery review questions are left in step 314, an assessment questionmay be allocated to the remaining progress slots in step 318. Thus,according to one example embodiment, assessment questions are providedonly where there are not enough review, progress, and assessmentquestions. Any incorrectly answered progress, assessment, or masteryreview questions may be designated as review questions in step 322 andadded to the pool of review questions in step 324.

A user may progress towards mastery of a designated word by correctlycompleting subsequent progress questions. In an embodiment, mastery ofthe designated word may occur by answering the progress questionscorrectly over more than one session. The number of progress questionsthat must be answered to achieve mastery of a word may vary by the wordand the number of sentences in the compiled index containing thatdesignated word. In an example embodiment, a user may need to answer atleast three progress questions to achieve mastery of a word. If threeprogress questions related to the designated word are not available,mastery of the word may be achieved by answering all the availablequestions. In an embodiment where multiple definitions exist, and thereexists questions for the multiple definitions, the system may beconfigured to output at least one question for each of the definitionsas progress questions in subsequent rounds. In this embodiment, no morethan five progress questions are needed to achieve mastery of thedesignated word, even if more than five definitions of the designatedword exist.

In an example embodiment of the present invention, after each round of10 questions, a user may be provided with a round summary providing asynopsis of the user's performance during the round. A user may benotified of words that the user answered correctly as well as words thatthe user incorrectly answered during the previous round.

In an example embodiment of the present invention, the system and methodmay be configured to present to the user a game based on the ActiveLearning List. For example, the system may select words that the userincorrectly answered in previous rounds. Words that may be incorporatedinto the game may include designated words from review questions thatthe user may be working on, from the Active Learning List. In an exampleembodiment, where the system generates words based on the ActiveLearning List, the system may be configured such that words that theuser has mastered or is denoted as progressing on are not used as thebasis for the generation of the game.

In an example embodiment, the game may be a synonym maze, where the usermay progress through a tile grid with a plurality of synonyms of one ofthe words of the Active Learning List (and/or including the word of theActive Learning List itself), such that each of the synonymous words isadjacent to at least one other of the synonymous words to form a pathbeginning at a first grid tile and ending at a second grid tile, e.g.,at an opposite edge of the grid tile than the first grid tile. The usermay connect from the beginning to the end of the maze by selecting thesynonymous words, as described in detail in U.S. Patent ApplicationSerial No. 13/075,863, entitled “System and Method for Advancement ofVocabulary Skills in a Game Environment,” and filed under AttorneyDocket No. 13212/10004, the entire content of which is herebyincorporated herein by reference.

In an alternative embodiment, the displayed game may be a matching gamecontaining a plurality of pairs of designated words, where a user mayselectively turn over two tiles, in an effort to find a matching pairfor a designated word. In an alternative embodiment, the designatedwords may be incorporated into a spelling bee, whereupon the user mayattempt to correctly spell a designated word. Other games incorporatingwords from the Active Learning List may be used, it being understoodthat the example games discussed do not represent an exhaustive list.

In an example embodiment of the present invention, the system and methodmay include a point system that may reward a user for achieving certainmilestones or achievements. A user may receive and accumulate points forselecting a correct answer to a question. In an embodiment, the numberof points earned may be dependent upon the question category asked. Forexample, a user may receive more points for correctly answering aprogress or mastery question, than for answering an assessment or reviewquestion. Points may also be received for different events, such asmastering a word, consecutively answering questions in a row, and/or forperforming well in an end of the round summary game, with the amount ofpoints given depending on the event accomplished. A user may alsoreceive special recognition (a badge) for the achievement of certainevents such as the mastering of a certain number of words, or themastering of a particular designated word.

In an example embodiment, the system and method may provide the userwith the option of receiving a hint for a question. A user may request ahint in step 226, 256, 266, and 277 (dependent on the question type),whereupon the user is given a choice of hints. Examples of hints mayinclude removing wrong answer choices or seeing the designated word usedin a sentence (if the question presented in not a fill-in sentence). Auser may select an answer based on the receipt and application of thehint. If a user chooses to receive a hint, the system may record thatthat the user requested a hint in steps 226, 256, 266, and 277. The usermay earn a smaller number of points for selecting the answer with theassistance of a hint. Even if a user selects the correct answer for theassessment question, the user may not be given credit for knowing theanswer and the question may be repeated in a subsequent round as areview question, because the user who requested a hint did not know theanswer without the hint.

In one example embodiment, the system may be configured such that thereis no option of hints for review, progress, or mastery review questions.Progress and mastery review questions may require demonstration that theuser fully understands and comprehends the designated word in order fora user to progress towards mastery of the word. Therefore, receiving ahint may indicate a lack of comprehension by the user. Review questionshave already been presented to a user, thus negating the need for a userto obtain a hint.

The presence of hints in the system may allow for the user to deduce theanswer without resorting to simple guessing. If the user guesses andcorrectly identifies the correct answer choice, the system mayerroneously determine that the user knew the answer to the presentedquestion, when the user did not. This may have the unwanted effect ofaltering the corresponding determined ability of the user, and the usermay receive questions that are outside of the user's ability range, thusnegating the purpose of the system to promote word comprehension.

In an example embodiment, the system may be configured such that, forthe same word (a) in the event that a user correctly answers a question,the system and method provides the user with only a simple explanationalluding to the definition of the designated word, and (b) in the eventthat the user incorrectly answers a question, the system outputs alonger explanation. The longer explanation (a “blurb”) may explain thenature of the designated word, including, but not restricted to, thecomplete definition of the word, the etymology of the word, synonyms andantonyms of the word, and the use of the word in an example sentence.

The above description is intended to be illustrative, and notrestrictive. Those skilled in the art can appreciate from the foregoingdescription that the present invention may be implemented in a varietyof forms, and that the various embodiments may be implemented alone orin combination. Furthermore, it will be appreciated that method stepsare not limited to the example sequence illustrated in the accompanyingflowcharts. Therefore, while the embodiments of the present inventionhave been described in connection with particular examples thereof, thetrue scope of the embodiments and/or methods of the present inventionshould not be so limited since other modifications will become apparentto the skilled practitioner upon a study of the drawings andspecification.

1. A computer-implemented method for generating vocabulary questions,the method comprising: obtaining, by a computer processor, a sentencefrom a database, the sentence having been assigned to one of a pluralityof quality categories; using the sentence, by the computer processor, togenerate a question concerning a word of the sentence, the questionbeing one of a plurality of question types selected based on therespective quality category to which the sentence has been assigned; andoutputting, by the computer processor, the question.
 2. The method ofclaim 1, further comprising outputting a set of selectable multiplechoice answers selectable for responding to the output question.
 3. Themethod of claim 2, wherein the set of multiple choice answers is a setof text strings, each including one or more words, which arealternatively selectable by the user for answering the generatedquestion.
 4. The method of claim 3, wherein only one of the set ofmultiple choice answers corresponds to the correct answer.
 5. The methodof claim 1, further comprising: retrieving text, by the computerprocessor, from an online content source; and analyzing at least onepunctuation mark in the retrieved text to determine at least oneboundary of the sentence in the retrieved text, the obtaining of thesentence being in accordance with the determined boundary.
 6. The methodof claim 5, wherein the retrieved text is timestamped, and the methodfurther comprises discarding text retrieved multiple times from a samesource within a predetermined period of time of the timestamp.
 7. Themethod of claim 5, further comprising: parsing the sentence, by thecomputer processor, to apply at least one marker for a definition forthe word according to use in the sentence, the word having multipledefinitions; and storing the sentence in the database, wherein thesentence is retrievable by the at least one definition marker.
 8. Themethod of claim 1, further comprising: assigning a priority to thesentence, by the computer processor, based on at least one of frequencyof use of words of the sentence and likelihood of appearance of thewords of the sentence on a standardized test.
 9. The method of claim 8,wherein a plurality of sentences, including the sentence to which thepriority was assigned, are reviewed in an order that is based on theassigned priority, during which review the plurality of sentences areassigned to respective ones of the quality categories.
 10. The method ofclaim 9, wherein the plurality of sentences are ranked by a difficultyor discrimination level.
 11. The method of claim 1, wherein the usingincludes modifying the sentence by removing the word to formulate thequestion.
 12. The method of claim 11, further comprising outputting aset of selectable multiple choice answers selectable for responding tothe output question, wherein similar words or synonyms are excluded fromthe set of multiple choice answers.
 13. The method of claim 1, whereinadditional questions are generated and output to the user, the questionand the additional questions forming a round of questions.
 14. Themethod of claim 13, wherein the question and additional questions aretailored to the user by asking the question and additional questions ina series of rounds, and the method further comprises tracking progressof the user based on a categorization of each question.
 15. The methodof claim 1, further comprising: removing boilerplate language from acorpus of text that includes the sentence.
 16. The method of claim 1,wherein an electronic dictionary provides at least one definition forthe word, and wherein, if the world has multiple definitions, a sense iscreated that indicates a meaning of the word in a context of thesentence.
 17. The method claim 1, further comprising: querying a corpusof words to determine a synonym or antonym of the word.
 18. The methodof claim 17, further comprising outputting a set of selectable multiplechoice answers selectable for responding to the output question, whereinone of the set of multiple choice answers corresponds to the synonym ofthe word and additional synonyms are excluded from the set of multiplechoice answers.
 19. The method of claim 17, further comprisingoutputting a set of selectable multiple choice answers selectable forresponding to the output question, wherein one of the set of multiplechoice answers corresponds to a definition of the word and similardefinitions and definitions of a synonym of the word are excluded fromthe set of multiple choice answers.
 20. The method of claim 17, furthercomprising outputting a set of selectable multiple choice answersselectable for responding to the output question, wherein one of the setof multiple choice answers corresponds to the antonym of the word andother antonyms of the word are excluded from the set of multiple choiceanswers.
 21. A computer-implemented method for generating vocabularyquestions, the method comprising: generating, by a computer processor, aquestion based on a stored sentence from a database, the questionconcerning a word in the stored sentence; outputting, by the computerprocessor, the question and a set of selectable multiple choice answers;and upon receiving a hint request by a user, outputting a sentence thatcontains the word.
 22. The method of claim 21, wherein the question ispresented again in a subsequent round after the user makes the hintrequest.
 23. The method of claim 21, further comprising logging receiptof an incorrect answer to the question, in response to the receipt ofthe hint request.
 24. A computer-implemented method for generatingvocabulary questions in an adaptive learning system, the methodcomprising: outputting, by a computer processor, questions in a seriesof rounds to a user, the questions being based on contained words insentences stored in a database, wherein each of the rounds contains adetermined number of question slots; allocating a set of the questionslots for questions previously missed by the user; distributingremaining questions slots to additional questions concerning words aboutwhich the user previously correctly answered questions; subsequentlydetermining whether any additional question slots are available; and ifthe determination is that there is one or more additional question slotsavailable, allotting the available question slots to new questionscontaining words about which the user has not been tested.
 25. Themethod of claim 24, wherein up to two questions slots are allocated forpreviously missed questions.
 26. The method of claim 24, wherein newquestions are only allotted in the question slots if the additionalquestions do not fill the remaining question slots.
 27. The method ofclaim 24, wherein the remaining question slots are distributed accordingto a percentage distribution to follow-up questions testing differentdefinitions of the words about which the user has previously correctlyanswered questions and mastery questions about words the user haspreviously mastered.
 28. The method of claim 27, wherein 90% of theremaining question slots are distributed to the follow-up questions and10% of the remainder is made up of the mastery questions.
 29. Anadaptive learning system for generating questions and multiple choiceanswers to assist in teaching word comprehension, the system comprising:a computer processor configured to: obtain at least one sentenceassigned to a respective one of a plurality of quality categories; usethe at least one sentence to generate a question concerning a word ofthe at least one sentence, the question being one of a plurality ofquestion types selected based on the respective quality category towhich the at least one sentence has been assigned; and output thequestion and a set of selectable multiple choice answers.
 30. The systemof claim 29, wherein the computer processor retrieves text from anonline content source, and analyzes at least one punctuation mark in theretrieved text to determine at last one boundary of the at least onesentence in the retrieved text, the obtaining being in accordance withthe determined at least one boundary.
 31. The system of claim 29,wherein the computer processor parses the at least one sentence to applyat least one marker for a definition for the word according to use inthe at least one sentence, and wherein the word has multipledefinitions.
 32. The system of claim 31, wherein the at least onesentence is retrievable based on the at least one definition marker. 33.The system of claim 29, wherein the computer processor assigns apriority to the at least one sentence based on at least one of frequencyof use of words of the sentence and likelihood of appearance of thewords of the sentence on a standardized test.
 34. The system of claim33, wherein a plurality of sentences, including the at least onesentence to which the priority was assigned, are reviewed in an orderthat is based on the assigned priority, during which the plurality ofsentences are assigned to respective ones of the quality categories. 35.A hardware computer-readable medium having stored thereon instructionsexecutable by a processor, the instructions which, when executed by theprocessor, cause the processor to perform a method for generatingvocabulary questions, the method comprising: obtaining a sentence from adatabase, the sentence having been assigned to one of a plurality ofquality categories; using the sentence to generate a question concerninga word of the sentence, the question being one of a plurality ofquestion types selected based on the respective quality category towhich the sentence has been assigned; and outputting the question.